Optimal Top-k Document Retrieval

Let D be a collection of D documents, which are strings over an alphabet of size σ, of total length n. We describe a data structure that uses linear space and and reports k most relevant documents that contain a query pattern P , which is a string of length p, in time O(p/ log σ n+k), which is optimal in the RAM model in the general case where lgD = Θ(logn), and involves a novel RAM-optimal suffix tree search. Our construction supports an ample set of important relevance measures, such as the number of times P appears in a document (called term frequency), a fixed document importance, and the minimal distance between two occurrences of P in a document. When lgD = o(log n), we show how to reduce the space of the data structure from O(n log n) to O(n(log σ+ logD+ log logn)) bits, and to O(n(log σ+ logD)) bits in the case of the popular term frequency measure of relevance, at the price of an additive term O(logε n log σ) in the query time, for any constant ε > 0. We also consider the dynamic scenario, where documents can be inserted and deleted from the collection. We obtain linear space and query time O(p(log logn)/ log σ n+logn+k log log k), whereas insertions and deletions require O(log n) time per symbol, for any constant ε > 0. Finally, we consider an extended static scenario where an extra parameter par(P, d) is defined, and the query must retrieve only documents d such that par(P, d) ∈ [τ1, τ2], where this range is specified at query time. We solve these queries using linear space and O(p/ log σ n + log n+ k logε n) time, for any constant ε > 0. Our technique is to translate these top-k problems into multidimensional geometric search problems. As an additional bonus, we describe some improvements to those problems.